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ABSTRACT 

In response to increasing assessment and 
accountability requirements, Mt. Hood Community College, in Oregon, 
developed an assessment matrix specifying major accountability 
requirements and established a calendar to help departments 
understand the necessary sequences for fulfilling these requirements. 
The matrix lists the various measures or indicators of effectiveness 
grouped among headings for student retention, outcomes, and success; 
special populations and disadvantaged; and other program and 
institutional indicators. For each indicator, the matrix also 
describes the purpose of assessment in terms of the following 
internal or external accountability sources: (1) accreditation; (2) 
internal program review; (3) the state quality assurance plan for all 
professional technical state approved programs; (4) state key 
effectiveness indicators; (5) the Community College Roundtable core 
indicators; and (6) the Joint Commission on Accountability Reporting. 
The calendar lists the major assessment pieces that need to be 
completed over a 2-year period. The integrated assessment approach 
has yielded positive results and improved communication among 
institutional researchers, administrators, and faculty at the 
college. Since the various accountability requirements contain 
inconsistencies in the measurement strategies, particularly in the 
area of student tracking, it is imperative to involve administrators 
early on in the process of developing an integrated approach to the 
assessment activities. Contains 12 references. A sample matrix and 
calendar are appended. (TGl) 
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Abstract 

Colleges and universities have been increasingly inundated by assessment and accountability 
requirements from numerous sources with inconsistent specifications. Often overlooked is how to 
effectively communicate assessment and accountability requirements to the campus community. This 
paper illustrates the effort of one college to address this issue by developing an assessment matrix that 
specifies major requirements and indicates overlapping and unique areas. Secondly, a calendar was 
created to help departments understand the sequence needed to complete various requirements in a 
timely manner. The institutional researchers worked closely with department administrators to develop 
specific step-by-step instructions and set realistic timelines for completion of assessment activities. Each 
of the assessment requirements included in the matrix and the reactions of various campus constituencies 
are described in the paper. 
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The Assessment Matrix; Communicating Assessment and 
Accountability Requirements to the Campus Community 

Colleges and universities have been increasingly inundated by assessment and accountability 
requirements from numerous sources with inconsistent specifications. There are federal and state 
government requirements, regional and specialized accreditation standards, and professional guidelines, 
in addition to the systems that colleges have in place for internal program review. Especially in the area of 
assessing student learning gains and outcomes, there ,s an ever increasing mountain of literature 
describing research results, how to conduct assessments, and the problems and issues involved iti 
assessment (Pascarella & Terenzini, 1991). 

Often overlooked, however, is how to effectively communicate assessment and accountability 
requirements to the larger campus community. Many administrators and faculty lack an overall 
understanding of the various requirements and how best to respond. As a result, departments often adopt 
a piecemeal approach, responding separately to each requirement, thus making more work for 
themselves. This situation can easily lead to a sense of being overwhelmed, resulting in poor assessment 
practices or complete failure to conduct assessments. It is suggested that perhaps a role for institutional 
researchers is to assist colleges in building on previous assessments and incorporating these into 
subsequent reporting requirements. 

The solution at this college was to develop a matrix that specifies the major assessment 
requirements and highlights areas of overlap and uniqueness. In addition, a calendar was created tc help 
departments understand the sequence needed to complete various requirements in a timely manner. We 
describe each of the assessment requirements identified in the matrix and the problems associated with 
developing a coordinated response. The reaction of various campus constituencies is discussed with a 
tentative appraisal of this particular technique. 

Although this study takes place at a community college, the requirements are common enough for 
any college or university to adapt. For example, almost all states now have performance reporting 
requirements related to student outcomes. Those using the model presented here would simply substitute 
the specific requirements from their state. The same reasoning holds true for accreditation; all regional 




4 



The Assessment Matrix 4 



accrediting bodies now include assessment of student learning gains and outcomes in their standards. 

We have also incorporated leading proposals for standardizing indicators of effectiveness such as being 
developed by the Joint Commission on Accountability Reporting (JCAR, 1995). 

Finally, the paper discusses some of the inconsistencies in definitions and indices related to the 
assessment of student outcomes. The lack of standardization creates both a problem in communicating 
requirements to the campus community and creating an additional workload for those who develop 
information systems to assist in meeting the various reporting requirements. It is especially important that 
presidents and other lead administrators understand the inconsistencies in reported information in terms 
of both resource implications for maintaining variable indices and in communicating with external 
constituencies. 

The Assessment Matrix 

Table 1 presents the assessment matrix. The first column lists the various measures or indicators 
of effectiveness grouped among headings for student retention/completion/success, special 
populations/disadvantaged, and so on. The remaining columns describe the assessment purpose based 
on a specific internal or external accountability source. 

Accreditation . Unique among the sources listed in Table 1, accreditation requirements do not 
mandate or specify measures. Guidelines do discuss areas of assessment such as retention, mid- 
program assessment, and capstone experience. However, colleges and universities are free to choose 
and design how those areas will be addressed in terms of measurement. “Standard V" refers to the 
guidelines prepared by the Northwest Association of Schools and Colleges (1994) dealing with the 
“educationf.1 program and its effectiveness.” 

Internal Program Review . The indicators listed here are derived from the college’s 
"program/function review” process, which is designed to serve both program improvement efforts and 
resource allocation decisions (Goal #3 Task Force, 1994). Thus, some of the measures deal with cost 
and other factors separate from the assessment of student outcomes. In the latter case there is an 
overlap with the other accountability sources. However, particular measurement strategies vary since the 
institution's requirements vary slightly from external requirements. 
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Perkins Quality Assurance . The indicators identified in this column are from the state plan (ODE, 

1 993) developed in response to requirements of the Carl D. Perkins Vocational and Applied Technology 
Education Act Amendments of 1990, Public Law 101- 392. The requirements cover all professional technical 
state approved programs. Programs must be reviewed annually using the indicators listed in Table 1, in 
addition to undergoing a comprehensive review every five years. In the comprehensive 'eview, college staff 
examine what are referred to as “process variables” which include quality of instruction, work experience 
opportunities for students, and other qualitative issues that presumably would affect student outcomes. 

State Key Effectiveness Indicators . These indicators were developed to respond to legislative 
accountability concerns (OCCS, 1992). The colleges and the state office have since designed and are 
implementing a statewide information system which includes monitoring these indicators. 

Core Indicators . The Community College Roundtable (1994) discussed thirteen measures related to 
the mission of a comprehensive community college. It is important to note that Core Indicators are a first cut, 
not a final and exhaustive treatment of the subject. The authors anticipate evolution and further refinement 
based on practice in the field. Nonetheless, this document represents a starting point for the development of 
a common framework from which community colleges can demonstrate effectiveness. 

Many of the indicators have been and continue to be used by colleges, especially in the area of 
institutional research. For example, placement rate in the workforce is commonly used in the evaluation of 
professional technical education programs, in this case, the indicator is also tied to federal reporting 
requirements as discussed with regard to the Perkins Act. Other indicators, such as demonstration of critical 
literacy and citizenship skills are employed by community colleges less frequently. Where such indicators 
are used, there may be a lack of documentation and/or evidence of the validity and reliability of the 
assessment instruments and methods. In general, the assessment of student learning is taking on more 
importance due to new and revised regional accreditation standards. Because of a relative lack of 
experience in this area, implementing formal assessment of student learning probably represents the most 
significant challenge to community colleges attempting to follow Core Indicators. 

In addition to a discussion of each indicator within the context of the community college mission, 
there is a "technical description" sectran dealing with measurement strategies and data sources for each 
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indicator. Again, this material is simply a starting point. Community college practitioners will need to conduct 
a systematic review of the indicators in relation to their current level of sophistication in assessment and 
evaluation, specific state accountability and regional accreditation requirements, and resources. 

Joint Commission on Accountability Reporting . The latest entrant into setting standards for 
assessing institutional effectiveness is the Joint Commission on Accountabiiity Reporting (JCAR, 1995). 
Invoiving the American Association of State Colieges and Universities, National Association of State 
Universities and Land-Grant Colleges, and American Association of Community Colleges, the goal of the 
commission is not only to identify key indicators but aiso to define very specific technicai definitions so that 
the resulting information wouid be comparable across institutions. The commission’s work represents an 
effort by the higher education community to take the lead in defining accountability requirements so as to 
prevent other groups, such as the federal government or the states, from mandating requirements. The 
difficulties in operationalizing the Student Right-to-Know Act has been a major influence in galvanizing action 
from the higher education community. However, it is too early to determine whether the work of the 
commission wili have a long-term impact. 

Schedule and Calendar for Assessment 

1 ie schedule and calendar for assessment can be found in Figure 1 . The schedule was created to 
help departments manage the timing of the activities needed to meet the assessment requirements. The 
calendar was also useful in depicting how the various assessment pieces fit together. 

The first column in the calendar lists the major assessment pieces that need to be completed over a 
two year period. The major assessment pieces are further divided into smaiier steps. Note that some of the 
same steps appear under mere than one assessment piece. To the right of the first column, a calendar 
divided by month and year runs across the top of the scheduie. Bars depicting the timeline for each 
assessment activity appear underneath the calendar heading. 

The assessment activity that spans the longest time period is accreditation. All other assessment 
activities within the two year period were integrated to meet the accreditation requirements and timeline. The 
hope is that as the departments complete the steps of the other assessments, they will be compiling data, 
program information, and narrative reports that will serve as the major components of the accreditation self- 
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study. With regard to the professional technical programs, the Perkins evaluation greatly overlaps with the 
internal program review requirements. The phase 1 data are the same as many of the Perkins outcome 
standards; phase 2 of the internal program review is based primarily on the Perkins process standards. The 
yearly area assessments that are part of the strategic planning process have also been incorporated into 
internal program review, and will be incorporated into the accreditation self-study. The ability to establish 
benchmarks and analyze trends over time is another benefit of planning for ongoing assessment activities. 
Inconsistencies in Definitions 

A sign’iicant complication in responding to various accountability requirements is the 
inconsistencies prescribed or suggested in measurement strategies. This problem is particularly 
pronounced in the area of student tracking, as the controversy and difficulties surrounding attempts to 
implement the Student Right-to-Know Act have demonstrated. 

Student retention and persistence rates are identified in accreditation guidelines, internal program 
review, by the Community College Roundtable (1994) and JCAR (1995). As is typical with accreditation 
guidelines, there are no prescribed procedures for measuring student retention. The other three are only 
consistent to the extent that student cohorts would include first-time, degree-seeking students. JCAR 
(1995) includes transfer students as a separate cohort while the other two are silent on transfer students. 
Of course, many community colleges do not identify transfer students at entry. Previous college 
experience is only relevant for students on financial aid or students applying for graduation and requesting 
transfer credits be applied toward their degree. 

There is no consistency in the definition of timelines for reporting persistence rates which spills 
over into the problem of calculating a graduation rate. The Community College Roundtable (1994, p. 17) 
notes “that community college students take as long as seven or eight years to earn a degree, the 
proposed indicator has no ‘endpoint’ time limit." As a result, both the Community College Roundtable and 
JCAR simply recommend annual reporting of persistence and graduation rates. For Perkins reporting, 
many community colleges track and follow-up on both graduates and what are termed '“early leavers” or 
"completers.” This would include students who have completed a substantial part of a program without a 
degree and have not returned to the college for a specified time period. 
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For accreditation, Perkins and internal program review persistence and graduation rates by 
program or field of study are required. However, beyond the generally accepted use of CIP (Classification 
of Instructional Programs) codes for categorizing students, the source material referenced in Table 1 does 
not address issues surrounding the calculation of persistence and graduation rates by student major. For 
example, how are students that change major dealt with in the tracking system? Even more fundamental 
is the problem of identifying a major for a particular student. Many colleges simply use student self- 
declared major, but such declarations may reflect student intent rather than actual enrollment. Dealing 
with this in a straightfon/irard manner is complicated by various financial aid and Perkins requirements 
(e.g., students must be in an approved program). 

In summary, it is extremely difficult for even a single institution to design a student tracking system 
that effectively responds to all the accountability requirements and guidelines. It is no wonder that the 
attempt to develop comparability across institutions, as in the case of Student Right-to-Know, has become 
such a daunting task. 

Reactions of Campus Constituencies 

Prior to the implementation of the assessment matrix, frustration with assessment practices was 
widespread across campus. Lack of communication between institutional researchers, administrators, 
and faculty created a number of obstacles to meeting the assessment requirements. 

One problem was that campus constituencies were unclear as to how the various assessment 
pieces fit together. Without a sense of how to integrate the assessment requirements to meet multiple 
purposes, they were overwhelmed. 

Second, the departments did not know what was expected of them in meeting the assessment 
requirements. The main complaint was that the materials outlining the assessment tasks were too vague 
and confusing. Often the associate deans aborted their attempts to respond to assessment requirements 
as they quickly became dismayed by the lack of specific guidelines provided for the task at hand. 

Third, the associate deans had little understanding of how to interpret data that were generated in 
the Office of Research and Planning and distributed for use in assessment activities. There were 
questions regarding how the various measures and standards were calculated and skepticism about 
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whether the data were accurate. Due to the communication breakdown on both sides, the unfortunate 
outcome was that the associate deans did not know what to make of the data and the Office of Research 
and Planning was not made aware of errors and discrepancies in the data that could be corrected. As a 
consequence of this climate of confusion, college constituents were at best frustrated and uncooperative 
participants in assessment activities. 

A final problem was that the lack of communication between the departments and the Office of 
Research and Planning led to further communication breakdown between the associate deans and 
program faculty. The associate deans, themselves frustrated and confused, had a difficult time explaining 
and justifying the assessment practices to their faculty. 

In response to this situation, the Office of Research and Planning redirected their efforts to focus 
on coordinating assessment activities so that they fit multiple purposes and communicating the integration 
of assessment requirements to campus constituents. In addition, the institutional researchers worked 
closely with the associate deans to develop step-by-step instructions for how to complete assessment 
activities, provide concrete examples based on program data, and create an activity calendar setting 
realistic timelines for completing the activities. 

So far, the reactions of the associate deans and faculty have been promising. Although they are 
still reluctant to begin the assessment activities, at least now they know how to approach the various 
demands and complete them in a timely manner. Because of the:'’ involvement in developing the 
assessment approach, the associate deans have a much deeper understanding of the assumptions 
underlying the data and what the figures mean. Another outcome of the new approach is that the 
associate deans are more likely to work with each other, sharing the strategies and information they have 
found useful in responding to the assessment requirements. This approach to assessment demands has 
also fostered better communication between the Office of Research and Planning, the associate deans, 
and the faculty. 

The activity calendar has helped campus constituents in two ways. First, it provides 
administrators and faculty with a way to pace their efforts and reduce the likelihood of missing windows of 
opportunity in which they can complete assessment activities. Second, it reiterates that there is overlap 
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and coordination between the assessment components. In other words, it enables constituents to see the 
“big picture”. 

Although the new approach to integrating assessment on campus has yielded positive results, 
there are still further improvements that need to be made. Constituents still experience frustration about 
finding the time to meet the requirements. Perhaps they do not yet believe that cooperation with the 
present assessment demands saves time in the long run. In addition, the associate deans and faculty 
now have a better understanding of the figures, so they are also better able to question the credibility of 
and point out the inconsistencies in the data. The positive side of this is that they are better able to bring 
these problems to the attention of the Office of Research and Planning. In turn, the Office of Research 
and Planning needs to respond promptly to these concerns. 

Implications for Institutional Research 

Given an institutional decision to implement any or a., of the indicators discussed above, what are the 
major elements of an action plan? What are the resource requirements? Assuming the decision involves all 
of the indicators, what is the order of priorities? Can the same system established to address one set of 
indicators be used to meet the other requirements? These are the key questions college staff must ask when 
designing the information system to support an institutional effectiveness program. 

The organizational structure of Core Indicators provides a logical basis for grouping the indicators. 
For example, there are three indicators under the “student progress" mission category: student goal 
attainment, persistence (fall to fall), and degree completion rates. All three of these measures would be 
derived from a student tracking system. There are numerous resources available to assist colleges in 
developing a student tracking system (Ewell, 1987; Palmer, 1990). At the same time, colleges must respond 
to the federal requirements contained in the Student Right-to-Know and Campus Security Act of 1990, the 
Perkins Act of 1990, and Higher Education Act amendments of 1992, all of which focus on student retention, 
graduation rates, and outcomes. Since there are no unresolved technical obstacles to implementing student 
tracking systems, the lack of one at any college today may simply reflect a lack of resources, or rrwre 
importantly, a failure to give such a system high priority. 
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Tile major complication with student tracking systems, especially at community colleges, is the 
diversity of student populations. Most student tracking systems are premised on continuous attendance and 
degree-seeking students. However, practitioners know that these assumptions do not apply to the majority of 
community college students. As a consequence, a considerable body of literature has developed that 
advocates the incorporation of student intent in the analysis of retention patterns (Walleri, 1990; Walleri, 
Seybert & Cosgrove, 1992; Palmer, 1993). 

Many states have or are in the process of developing shared information systems which facilitate 
the collects jn of follow-up information on employment and transfer (Walleri, 1990; JCAR, 1995). By 
matching or: .^udent SSN, colleges can exchange information on transfer among themselves as well as 
with other state agencies that collect information on employment, wages and so on. 

Once the decision to make student-tracking a priority has been made, regardless of the details of 
setting up such a system, it is imperative to involve administrators early on in the process of developing an 
integrated a, proach to the assessment activities that will require their participation. The more that they 
understand the integration of various assessment activities and participate in creating step-by-step 
procedures and realistic timelines, the more likely they are to cooperate in meeting the assessment 
requirements. Furthermore, this process improves communication and understanding of the 
accountability demands and the assessment results on the part of all constituents. 
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Figure Caption 

. Figure 1. Timelines for multiple assessment activities. 
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